A file server is a network-connected storage server that stores and manages shared files in a set of storage devices (e.g., disk drives) on behalf of one or more clients. The disks within a file system are typically organized as one or more groups of Redundant Array of Independent/Inexpensive Disks (RAID). One configuration in which file servers can be used is a network attached storage (NAS) configuration. In a NAS configuration, a file server can be implemented in the form of an appliance that attaches to a network, such as a local area network (LAN) or a corporate intranet. An example of such an appliance is any of the Filer products made by Network Appliance, Inc. in Sunnyvale, Calif.
Another specialized type of network is a storage area network (SAN). A SAN is a highly efficient network of interconnected, shared storage devices. Such devices are also made by Network Appliance, Inc. One difference between NAS and SAN is that in a SAN, the storage appliance provides a remote host with block-level access to stored data, whereas in a NAS configuration, the file server normally provides clients with file-level access to stored data.
Current file server systems used in NAS environments are generally packaged in either of two main forms: 1) an all-in-one custom-designed system that is a standard computer with built-in disk drives all in a single chassis (enclosure); or 2) a modular system in which one or more sets of disk drives, each in a separate chassis, are connected to an external file server “head” in another chassis.
In this context, the term “head” means all of the electronics, firmware and/or software (the “intelligence”) that is used to control access to storage devices in a storage system; it does not include the disk drives themselves. In a file server, the head normally is where all of the “intelligence” of the file server resides. Note that a “head” in this context is not the same as, and is not to be confused with, the magnetic or optical head used to physically read or write data to a disk.
In a modular file server system, the system can be built up by adding multiple chassis in some form of rack and then cabling the chassis together. The disk drive enclosures are often called “shelves” and, more specifically, “just a bunch of disks” (JBOD) shelves. The term JBOD indicates that the enclosure essentially contains only physical storage devices and little or no electronic “intelligence.” Some disk drive enclosures include one or more RAID controllers, but such enclosures are not normally referred to as “JBOD” due to their greater functional capabilities. A modular file server system is illustrated in FIG. 1 and is sometimes called a “rack and stack” system. In FIG. 1, a file server storage server head 1 is connected by external cables to multiple disk shelves 2 mounted in a rack 3. The file server storage server head 1 enables access to stored data by one or more remote client computers (not shown) that are connected to the storage server head 1 by external cables.
Enterprise class storage appliances (e.g., filers) have unique and class-defining characteristics of their own, such as the stringent requirement to be up and running almost hundred percent of the time. In order to increase the amount of uptime, these appliances need to be resistant to certain types of errors. One conventional method employed to ensure that appliances keep serving data, despite catastrophic types of errors, is to provide multiple data paths. Provisioning of multiple data paths can be accomplished at different system level components. For example, in conventional storage appliances in highly available (HA) configurations, dual heads are provided to insulate against a broad range of motherboard level errors that would cripple a single head, but the remaining head would assume the identity of the fallen head, still serving data to the user of the fallen head.
Small computer system interface (SCSI), which has been inexistence for well over twenty years, has traditionally been used in a wide variety of enterprise class storage solutions. This multi-layered architecture is extensively used and acknowledged by industry leaders as a stable, feature rich extensible IO interconnect. The SCSI Architectural Model (SAM), has proven its reliability over four generations of parallel physical layers and asynchronous and synchronous transfer speeds, as well as being adopted by other popular physical interconnects, such as Fibre Channel and ATAPI.
The latest generation of the implementation of SAM is serial attached SCSI (SAS), succeeding over Ultra 320 parallel SCSI. SAS eliminated parallel bus issues such as multi-drop, skew, limited distance, small form factor, and introduced some of the well defined advantages of using a high speed serial point-to-point link, such as elimination of skew and cross talk, fewer wires, low power etc. SAS also became the unifying point of the two dominant disk command protocols of our time, SCSI and Advanced Technology attachment (ATA), where SAS protocol ensured that SATA drives could be operated in a SAS domain.
SAS also addressed certain other deficiencies of SCSI implementation, an important one being the maximum number of devices that can be connected to form a SCSI domain, by introducing a different device addressing scheme and a routing scheme. The maximum number of devices in a SCSI domain, which used to be 16 in parallel SCSI and 127 in Fibre Channel, has grown out of proportion in SAS, to well over a thousand devices as a theoretical maximum. This property coupled with the point-to-point nature in SAS links, gave rise to the concept of expanders in SAS.
As the name implies, expanders typically increase the number of devices that can be physically connected to a SAS domain. Most SAS host bus adapters found in the market today contain 4 or 8 physical links (“phys”), limiting the number of devices that can be connected to them directly to 4 or 8 respectively. But with the help of expander, which can contain, for example, 12 or 24 or 36 phys, and their ability to be deeply cascaded, an extremely large number of drives can now be connected to a storage server. SAS also brings the added degree of resilience by allowing more than one phy in a wide port configuration to connect expanders to expander or expanders to host bus adapters. In such a connection, e.g., 4 phys wide, even if 3 phys fail, the remaining phy can still provide access to all of the devices downstream of that connection.
The presence of multiple SAS expanders containing system enclosure services (SES) processors presents the problem of having to synchronize the states of the firmware in both expanders to present a uniform view of the enclosure to the external world. The “state” of firmware, or state information, in each expander can include, for example, data needed to form SAS addresses to SATA drives, data read from sensors, and data needed to serve certain diagnostic pages as defined by the SES specification. Current solutions of SAS based appliances use an Inter-Integrated Circuit (I2C) connection between the expanders to allow them to communicate among themselves. I2C connections, however, are known for failures and hang-ups, preventing any kind of communication from taking place between the devices. Although certain precautions can be taken to minimize the possibility of an appliance failing in the case of a failure of the I2C connection, there is no conventional method known that can be used to enable expanders to communicate with each other to synchronize their state upon such failure.